Data description.

# load R libraries
if (!require(tidyverse)) install.packages("tidyverse", repos = "http://cran.us.r-project.org")
if (!require(DataExplorer)) install.packages("DataExplorer", repos = "http://cran.us.r-project.org")
if (!require(htmltools)) install.packages("htmltools", repos = "http://cran.us.r-project.org")
if (!require(ggstatsplot)) install.packages("ggstatsplot", repos = "http://cran.us.r-project.org")
if (!require(plotly)) install.packages("plotly", repos = "http://cran.us.r-project.org")
if (!require(rmdformats)) install.packages("rmdformats", repos = "http://cran.us.r-project.org")

#load Python modules
import numpy as np
import pandas as pd
import matplotlib
import matplotlib.pyplot as plt
import sweetviz as sv

import xgboost as xgb
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier

from sklearn.model_selection import train_test_split
from sklearn.feature_selection import RFE
from sklearn.model_selection import cross_val_predict, cross_val_score

from sklearn.metrics import confusion_matrix, classification_report, precision_recall_fscore_support, roc_curve, roc_auc_score, accuracy_score, recall_score, precision_score

from sklearn import metrics
from sklearn.metrics import confusion_matrix

#Import dataset
dataset = pd.read_csv("dataset/bank-additional-full.csv", sep = ";")

dataset['y'] = dataset['y'].apply(lambda x: 0 if x =='no' else 1)
dataset.rename(columns = {"y" : "deposit"}, inplace = True)

dataset.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 41188 entries, 0 to 41187
Data columns (total 21 columns):
 #   Column          Non-Null Count  Dtype  
---  ------          --------------  -----  
 0   age             41188 non-null  int64  
 1   job             41188 non-null  object 
 2   marital         41188 non-null  object 
 3   education       41188 non-null  object 
 4   default         41188 non-null  object 
 5   housing         41188 non-null  object 
 6   loan            41188 non-null  object 
 7   contact         41188 non-null  object 
 8   month           41188 non-null  object 
 9   day_of_week     41188 non-null  object 
 10  duration        41188 non-null  int64  
 11  campaign        41188 non-null  int64  
 12  pdays           41188 non-null  int64  
 13  previous        41188 non-null  int64  
 14  poutcome        41188 non-null  object 
 15  emp.var.rate    41188 non-null  float64
 16  cons.price.idx  41188 non-null  float64
 17  cons.conf.idx   41188 non-null  float64
 18  euribor3m       41188 non-null  float64
 19  nr.employed     41188 non-null  float64
 20  deposit         41188 non-null  int64  
dtypes: float64(5), int64(6), object(10)
memory usage: 6.6+ MB

dataset['pdays'] = dataset['pdays'].apply(lambda x: 0 if x ==999 else x)

EDA analysis:

From the target variable, we can see that 88.73 % of the customers haven’t subscribed to the financial product offered, therefore we have an imbalanced dataset. This situation will be refectled in the train, validation and test sets when modelling. For that we will use either the undersampling or oversampling techniques.

From the correlation plot we can observe that there are important correlations in several characteristics with respect to the variable “deposit” as well as between them. The above correlation matrix was plotted with all variables. Clearly, “campaign outcome” has a strong correlation with “duration,” a moderate correlation with “previous contacts,” and mild correlations between “balance,” “month of contact” and “number of campaign.”

grouped_gghistostats(
    data = dataset,
    x = age,
    grouping.var = deposit, # grouping variable
    normal.curve = TRUE, # superimpose a normal distribution curve
    normal.curve.args = list(color = "red", size = 1),
    ggtheme = ggthemes::theme_tufte(),
    plotgrid.args = list(nrow = 1),
    ggstatsplot.layer = FALSE,
    ggplot.component = list(theme(text = element_text(size = 6.3))),
    annotation.args = list(title = "Age distribution by deposit")
)

In the age variable, we observe that age is not an element that has much difference between customers who took deposits from those who did not, its average is around 40 years. However, both are 2 different groups statistically speaking due to the low p-value in the t test. The only remarkable difference we can highlight, is the fact that most of the old people subscribed to a deposit.

ggbarstats(data = dataset, x = education, y = deposit, title = "Education by deposit subscription", 
    legend.title = "Educational level", ggtheme = hrbrthemes::theme_ipsum_pub())

Education shows a difference between the different levels. For example, some types of clients present an efficiency of 13.72% (university students) while those with basic levels of studies do not reach 9% in some cases. We could say that we should aim to offer this product to clients who have college, professional or high school levels.

In the case of type of work, retirees, students, unemployed and management positions are those who lead with the best results to offer the financial product.

Regarding marital status, we could infer that single clients are a bit more sensitive to acquire the offer of term deposits.

The month variable is a good indicator. Note that the number of contacts and their efficiency varies strongly from month to month. For example, in March we obtained 50% efficiency with very few contacts made (only 500), however, in May 14 thousand contacts were made with only an efficiency of 6.4%

Regarding the variable pdays we can say that most of the clients were contacted for the first time.

ggbarstats(data = dataset, x = poutcome, y = deposit, title = "Outcome of the previous marketing campaign by current deposit subscription", 
    legend.title = "Educational level", ggtheme = hrbrthemes::theme_ipsum_pub())

Out of the people that subscribed for a deposit, only 19% of these customers had a previous deposit (sucessfull result) in the previous campaign.

As for the next section regarding the data pre-processing there is no need to impute the data since we don’t have missing values, regarding the outliers, we can see few outliers in the variable “age” and we will accept them since there are no regulations regarding the age of a customer to subscribe to a term deposit. If this case study would be credit-risk related, then we would have to discard these outliers or manipulate them.

One-Hot Encoding of categorical variables

Most of our categorical data are variables that contain label values rather than numeric values. The number of possible values is often limited to a fixed set. The problem when modeling using categorical data is that some algorithms can work with categorical data directly, and a preliminary transformation of the variables has to be done prior the modeling process.

For example, a decision tree can be trained directly from categorical data with no data transform required (this depends on the specific implementation).

To this, many machine learning algorithms cannot operate on label data directly. They require all input variables and output variables to be numeric. In general, this is mostly a constraint of the efficient implementation of machine learning algorithms rather than hard limitations on the algorithms themselves.

The main idea is to split the column which contains categorical data to many columns depending on the number of categories present in that column. Each column contains “0” or “1” corresponding to which column it has been placed.

First: we create two data sets for numeric and non-numeric data

numerical = dataset.select_dtypes(exclude=['object'])
categorical = dataset.select_dtypes(include=['object'])

Second: One-hot encode the non-numeric columns

onehot = pd.get_dummies(categorical)

Third: Union the one-hot encoded columns to the numeric ones

df = pd.concat([numerical, onehot], axis=1)

Fourth: Print the columns in the new data set

glimpse(py$df)

Rows: 41,188
Columns: 64
$ age                           <dbl> 56, 57, 37, 40, 56, 45, 59, 41, 24, 25, ~
$ duration                      <dbl> 261, 149, 226, 151, 307, 198, 139, 217, ~
$ campaign                      <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1~
$ pdays                         <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0~
$ previous                      <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0~
$ emp.var.rate                  <dbl> 1.1, 1.1, 1.1, 1.1, 1.1, 1.1, 1.1, 1.1, ~
$ cons.price.idx                <dbl> 93.994, 93.994, 93.994, 93.994, 93.994, ~
$ cons.conf.idx                 <dbl> -36.4, -36.4, -36.4, -36.4, -36.4, -36.4~
$ euribor3m                     <dbl> 4.857, 4.857, 4.857, 4.857, 4.857, 4.857~
$ nr.employed                   <dbl> 5191, 5191, 5191, 5191, 5191, 5191, 5191~
$ deposit                       <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0~
$ job_admin.                    <int> 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0~
$ `job_blue-collar`             <int> 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0~
$ job_entrepreneur              <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0~
$ job_housemaid                 <int> 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1~
$ job_management                <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0~
$ job_retired                   <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0~
$ `job_self-employed`           <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0~
$ job_services                  <int> 0, 1, 1, 0, 1, 1, 0, 0, 0, 1, 0, 1, 0, 0~
$ job_student                   <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0~
$ job_technician                <int> 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0~
$ job_unemployed                <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0~
$ job_unknown                   <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0~
$ marital_divorced              <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1~
$ marital_married               <int> 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0~
$ marital_single                <int> 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 0~
$ marital_unknown               <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0~
$ education_basic.4y            <int> 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1~
$ education_basic.6y            <int> 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0~
$ education_basic.9y            <int> 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0~
$ education_high.school         <int> 0, 1, 1, 0, 1, 0, 0, 0, 0, 1, 0, 1, 1, 0~
$ education_illiterate          <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0~
$ education_professional.course <int> 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0~
$ education_university.degree   <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0~
$ education_unknown             <int> 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0~
$ default_no                    <int> 1, 0, 1, 1, 1, 0, 1, 0, 1, 1, 0, 1, 1, 1~
$ default_unknown               <int> 0, 1, 0, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0~
$ default_yes                   <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0~
$ housing_no                    <int> 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 1, 0~
$ housing_unknown               <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0~
$ housing_yes                   <int> 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 1~
$ loan_no                       <int> 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1~
$ loan_unknown                  <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0~
$ loan_yes                      <int> 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0~
$ contact_cellular              <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0~
$ contact_telephone             <int> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1~
$ month_apr                     <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0~
$ month_aug                     <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0~
$ month_dec                     <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0~
$ month_jul                     <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0~
$ month_jun                     <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0~
$ month_mar                     <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0~
$ month_may                     <int> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1~
$ month_nov                     <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0~
$ month_oct                     <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0~
$ month_sep                     <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0~
$ day_of_week_fri               <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0~
$ day_of_week_mon               <int> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1~
$ day_of_week_thu               <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0~
$ day_of_week_tue               <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0~
$ day_of_week_wed               <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0~
$ poutcome_failure              <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0~
$ poutcome_nonexistent          <int> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1~
$ poutcome_success              <int> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0~

With this method we end up with a larger dataframe of 64 columns.

df.shape

(41188, 64)

Creation of Training, Validation and Test datasets

In any machine learning project, after a EDA a common practice is to split the dataset into training, test and validation (if applies). For this we set a seed (123) for sampling reproducibility and split the one-hot encoded dataset in a training, validation and test set, by using pandas and numpy. The training set will contain 70% of the data, 15% for validation and the remaining 15% for our test set.

# We create the X and y data sets
X = df.loc[ : , df.columns != 'deposit']
y = df[['deposit']]

# Create training, evaluation and test sets
X_train, test_X, y_train, test_y = train_test_split(X, y, test_size=.3, random_state=123)
X_eval, X_test, y_eval, y_test = train_test_split(test_X, test_y, test_size=.5, random_state=123)

In order to check how imbalanced is our training dataset in terms of our target variable “deposit,” we run the following code to calculate the percentage of customers that did not subscribe to a term deposit in the train set.

# percentage of defaults and non-defaults in the training set
round(y_train['deposit'].value_counts()*100/len(y_train['deposit']), 2)

0    88.75
1    11.25
Name: deposit, dtype: float64

We find that 88.75% of the customer did not subscribe to a term deposit, and 11.25% got this financial product. For modeling purposes our dataset cannot be imbalanced as it would bias our estimations since many algorithms assume a balanced or closely balanced dataset. In the next section we will proceed with a technique that will allow us to balance our training set.

Balancing dataset

Imbalanced data typically refers to a problem with classification problems where the classes are not represented equally. Most classification data sets do not have exactly equal number of instances in each class, but a small difference often does not matter, however, our dataset is imbalanced.

Our imbalanced dataset is not adequate for predictive modeling, as mentioned above, most of the machine learning algorithms used for classification were designed around the assumption of an equal number of examples for each class. This results in models that have poor predictive performance, specifically for the minority class. This is a problem because typically, the minority class is more important and therefore the problem is more sensitive to classification errors for the minority class than the majority class.

For balancing our dataset we will use the undersampling technique that consists in sampling from the majority class in order to keep only a part of these points. This will reduce the number of rows of our dataset, however, we can afford to apply such method because our training set is quite large.

First we create data sets for deposits and no-deposits:

X_y_train = pd.concat([X_train.reset_index(drop = True), y_train.reset_index(drop = True)], axis = 1)
count_no_deposit, count_deposit = X_y_train['deposit'].value_counts()
no_deposit = X_y_train[X_y_train['deposit'] == 0]
deposit = X_y_train[X_y_train['deposit'] == 1]

Second we undersample the no-deposits

no_deposit_under = no_deposit.sample(count_deposit)

Third, we concatenate the undersampled nondefaults with defaults

train_balanced = pd.concat([no_deposit_under.reset_index(drop = True), deposit.reset_index(drop = True)], axis = 0)

Lastly, we check the proportion of deposit and no deposits in our target variable:

round(train_balanced['deposit'].value_counts()*100/len(train_balanced['deposit']), 2)

1    50.0
0    50.0
Name: deposit, dtype: float64

We get a balanced training dataset with 50% of customers that subscribed to a term deposit, and another 50% that did not. However, this undersampled but balanced dataset has now 6488 rows.

From our balanced train dataset we set our X_train feature matrix that contains all independent variables by running the following code:

X_train = train_balanced.loc[ : , train_balanced.columns != 'deposit']
y_train = train_balanced[['deposit']]

Logistic Regression Model

Logistic regression models predicts the probability of the default class. In our case, this model will predict the probability of a customer subscribing to a term deposit.

We start by training the logistic regression model on the training data.

clf_logistic = LogisticRegression(max_iter = 100000).fit(X_train, np.ravel(y_train))

Based on the trained model, we predict the probability that a customer has to subscribing to a term deposits using validation data.

preds = clf_logistic.predict_proba(X_eval)

The function used in the previous chunk of code predict_proba provides probabilities for in a range of (0,1) including float numbers. The first column refers to the probability for a customer of not getting a term deposit, and the second column is the probability of subscribing to a term deposit. Now, we create a dataframe of predictions of subscribing to a term deposit, and the true values of people that subscribed to a term deposit:

preds_df = pd.DataFrame(preds[:,1], columns = ['prob_accept_deposit'])
true_df = y_eval
pred_comparison = pd.concat([true_df.reset_index(drop = True), preds_df], axis = 1)
pred_comparison.head(10)

   deposit  prob_accept_deposit
0        0             0.036086
1        0             0.039713
2        0             0.794001
3        0             0.054322
4        0             0.012686
5        0             0.355502
6        0             0.025778
7        0             0.021069
8        0             0.069364
9        1             0.165333

We are interested in checking the classification report of this model. For this, we reassign the probability of accepting a deposit based on the threshold 0.5 which is the middle point between 0 and 1, this is a common approach is many other algorithms. In other words, any estimated probability higher than 0.5 will be assigned as a deposit (1), otherwise as a no-deposit (0).

preds_df['prob_accept_deposit'] = preds_df['prob_accept_deposit'].apply(lambda x: 1 if x > 0.5 else 0)

We can slightly compare how differ the estimations with the real values by the difference in deposits.

Count of estimated deposits by the logistic model:

print(preds_df['prob_accept_deposit'].value_counts())

0    4760
1    1418
Name: prob_accept_deposit, dtype: int64

Count of real deposits in our test set:

print(true_df['deposit'].value_counts())

0    5496
1     682
Name: deposit, dtype: int64

Choosing the right metric is crucial while evaluating machine learning (ML) models. Various metrics are proposed to evaluate ML models in different applications

By the nature of this case study we are having a classification problem. Therefore we will choose as our metric for model performance Recall (aka Sensitivity, TPR or True Positive Rate) which is defined as the fraction of samples from a class which are correctly predicted by the model.

The Recall metric provides us with the answer to a the question “Of all of the positive samples, what proportion did I predict correctly?” It concentrates on the false negatives (FN) and are observations that our algorithm missed. The lower the number of FN is, the better prediction power of our model. In this case study we have 2 classes in our target variable, whether a customer subscribes to a term deposit or not. To this we will analyse the same metric for both classes.

\(Recall(Deposit) = \frac{True Positives}{True Positives + False Negatives}\)

\(Recall(No-Deposit) = \frac{True Negatives}{True Negatives+ False Positives}\)

Another important metric that will be also analysed but not taken in consideration when choosing the models is the Accuracy, this is perhaps the simplest metrics one can imagine, and is defined as the number of correct predictions divided by the total number of predictions.

\(Accuracy = \frac{True Positives + True Negatives}{True Positives + False Positives + True Negatives + False Negatives}\)

In order to check the performance of our model it is necessary to check the classification report, by running the following chunk of code:

target_names = ['No-deposit', 'Deposit']
print(classification_report(y_eval, preds_df['prob_accept_deposit'], target_names=target_names))

              precision    recall  f1-score   support

  No-deposit       0.99      0.85      0.91      5496
     Deposit       0.43      0.90      0.58       682

    accuracy                           0.86      6178
   macro avg       0.71      0.88      0.75      6178
weighted avg       0.92      0.86      0.88      6178

We check the accuracy score the model as follows, although this is not our metric of interest. We check this value by observing the above table or by running the following chunk of code:

print(clf_logistic.score(X_eval, y_eval).round(2))

0.86

It means that this model correctly predicts 85% of the classes. Finally, we check the confusion matrix which is a table with 4 different combinations of predicted and actual values

Where:

TN = True Negatives

TP = True Positives

FN = False Negatives

FP = False positives

# Print the confusion matrix
matrix = confusion_matrix(y_eval,preds_df['prob_accept_deposit'])
print(matrix)

[[4692  804]
 [  68  614]]

TN = 4696

TP = 611

FN = 71

FP = 800

We are interested to evaluate our model based on the metric recall aka true positive rate for subscribing to a term deposits which can visualised in the classification report:

recall_log_reg_1 = round(matrix[1][1]/(matrix[1][1]+matrix[1][0]), 2)
print(recall_log_reg_1)

0.9

Also, we want to improve this metric by maximizing in terms of accuracy. As seen in before, the cut-off point for assigning the categories from the predictions was 0.5. The cut-off point is the point that will indicate whether a customer with certain characteristics will subscribe to a term deposit. If the probability becomes more than the cut-off point, the customer will be in the class of “Deposit,” otherwise will be in the class of “No-deposit.”

We can however set an optimal threshold for the classification and improve our recall metric. Before proceeding with this we must reset the preds_df dataframe with the original predicted probabilities, and overwrite those that resulted from the previous arbitrary cut-off

preds_df = pd.DataFrame(preds[:,1], columns = ['prob_accept_deposit']).reset_index(drop = True)

First we run a for loop that evaluates the model’s performance with different probability cut-offs points, from 0 to 1 by increments of 0.001.

numbers  = [float(x)/1000 for x in range(1000)]
for i in numbers:
    preds_df[i]= preds_df.prob_accept_deposit.map(lambda x: 1 if x > i else 0)
preds_df.head(5)

   prob_accept_deposit  0.0  0.001  0.002  ...  0.996  0.997  0.998  0.999
0             0.036086    1      1      1  ...      0      0      0      0
1             0.039713    1      1      1  ...      0      0      0      0
2             0.794001    1      1      1  ...      0      0      0      0
3             0.054322    1      1      1  ...      0      0      0      0
4             0.012686    1      1      1  ...      0      0      0      0

[5 rows x 1001 columns]

Then we calculate the metrics: accuracy sensitivity, and recalls for deposit and no deposit for various probability cutoffs.

cutoff_df = pd.DataFrame( columns = ['prob','accs','def_recalls','nondef_recalls'])
for i in numbers:
    cm1 = metrics.confusion_matrix(true_df, preds_df[i])
    total1=sum(sum(cm1))
    accs = (cm1[0][0]+cm1[1][1])/total1
    
    def_recalls = cm1[1][1]/(cm1[1][1]+cm1[1][0])
    nondef_recalls = cm1[0][0]/(cm1[0][0]+cm1[0][1])
    cutoff_df.loc[i] =[ i ,accs,def_recalls,nondef_recalls]
print(cutoff_df.head(5))

        prob      accs  def_recalls  nondef_recalls
0.000  0.000  0.110392          1.0        0.000000
0.001  0.001  0.110392          1.0        0.000000
0.002  0.002  0.110392          1.0        0.000000
0.003  0.003  0.110554          1.0        0.000182
0.004  0.004  0.111525          1.0        0.001274

Now we are able to choose the best cut-off based on the trade-off between deposit recall and no-deposit recall.

cutoff_df <- py$cutoff_df

names(cutoff_df)[names(cutoff_df) == "prob"] <- "Probability cut-off"
names(cutoff_df)[names(cutoff_df) == "accs"] <- "Accuracy"
names(cutoff_df)[names(cutoff_df) == "def_recalls"] <- "Deposit Recall"
names(cutoff_df)[names(cutoff_df) == "nondef_recalls"] <- "No-deposit Recall"

cutoff_df <- cutoff_df %>%
    gather(key = "metric", value = "value", -`Probability cut-off`)

ggplot(cutoff_df, aes(x = `Probability cut-off`, y = value, color = metric)) + geom_line(aes(linetype = metric)) + 
    ggtitle("Accuracy, Deposit Recall and No-deposit Recall") + scale_fill_discrete(name = "Metrics", 
    labels = c("Accuracy", "Deposit Recall", "No-deposit Recall"))

The optimal cut-off point is the following:

cutoff_df["diff"] = abs(cutoff_df.def_recalls - cutoff_df.nondef_recalls)
best_threshold = cutoff_df["prob"].loc[cutoff_df["diff"] == min(cutoff_df["diff"])]
best_threshold = best_threshold.iloc[0]
print(best_threshold)

0.555

Now we can implement the optimal threshold to the model. Again, we calculate the probability predictions from the model, then we create a dataframe with such predictions.

preds = clf_logistic.predict_proba(X_eval)

preds_df = pd.DataFrame(preds[:,1], columns = ['prob_accept_deposit'])
true_df = y_eval

Then we reassign the probability of accepting a deposit based on the optimal threshold.

preds_df['prob_accept_deposit'] = preds_df['prob_accept_deposit'].apply(lambda x: 1 if x > best_threshold else 0)

We can slightly compare how differ the estimations with the real values by the difference in deposits.

Count of estimated deposits by the logistic model:

print(preds_df['prob_accept_deposit'].value_counts())

0    4876
1    1302
Name: prob_accept_deposit, dtype: int64

Count of real deposits in our test set:

print(true_df['deposit'].value_counts())

0    5496
1     682
Name: deposit, dtype: int64

For further information it is necessary to check the classification report:

target_names = ['No-deposit', 'Deposit']
print(classification_report(y_eval, preds_df['prob_accept_deposit'], target_names=target_names))

              precision    recall  f1-score   support

  No-deposit       0.98      0.87      0.92      5496
     Deposit       0.46      0.87      0.60       682

    accuracy                           0.87      6178
   macro avg       0.72      0.87      0.76      6178
weighted avg       0.92      0.87      0.89      6178

By setting this new cut-off our recall metric is balanced in both classes and the model improves the correctness of classification in each of the classes.

We check the confusion matrix and compare with the previous one

# Print the confusion matrix
matrix_2 = confusion_matrix(y_eval,preds_df['prob_accept_deposit'])
print(matrix_2)

[[4788  708]
 [  88  594]]

TN = 4778

TP = 593

FN = 89

FP = 718

Now we check the accuracy after assigning the values with the new cut-off point.

accuracy_log_reg_1 = round((matrix_2[0][0]+matrix_2[1][1])/sum(sum(matrix_2)), 3)
print(accuracy_log_reg_1)

0.871

There is a considerable improvement in the accuracy.

We are interested to evaluate our model based on the metric recall aka true positive rate for subscribing to a term deposits:

recall_deposit_log_reg_1 = round(matrix_2[1][1]/(matrix_2[1][1]+matrix_2[1][0]), 2)
print(recall_deposit_log_reg_1)

0.87

Now we proceed to calculate the Area Under Curve (AUC) score: stands for “Area under the ROC Curve.” The AUC measures the entire two-dimensional area underneath the entire ROC curve and allows comparison of classifiers by comparing the total area underneath the line produced on the ROC curve. AUC ranges in value from 0 to 1. A model whose predictions are 100% wrong has an AUC of 0.0, and one whose predictions are 100% correct has an AUC of 1.0.

The Receiver Operating Characteristic (ROC) Curve: is a two-dimensional graph that depicts trade-offs between benefits (true positives) and costs (false positives). It displays a relation between sensitivity and specificity for a given classifier. The TPR (True Positive Rate) is plot on the Y axis and the FPR (False Positive Rate) on the X axis, where the TPR is the percentage of true positives (to the sum of true positives and false negatives and the FPR is the percentage of false positives (to the sum of false positives and true negatives. A ROC curve examines a single classifier over a set of classification thresholds.

prob_deposit_log_reg_1 = preds[:, 1]
auc_log_reg_1 = round(roc_auc_score(y_eval, prob_deposit_log_reg_1), 3)
print(auc_log_reg_1)

0.939

Regularized Logistic Regression Model

In this section we use the same model (logistic regression), however, this time we use regularization techniques. Regularization techniques work by limiting the capacity of models (such as logistic regression) by adding a parameter norm penalty $\lambda$ to the objective function. As follows:

Generally we trade off some bias to get lower variance, and lower variance estimators tend to overfit less. However, our Ridge Regression (aka L2-norm penalty) is an assumption about the function we’re fitting (we’re assuming that it has a small gradient). In general, when we trade off bias for lower variance, it’s because we’re biasing towards the kind of functions we want to fit.

Our logistic regression model uses the optimisation algorithm Stochastic Average Gradient (SAG), also we set max_iter = 10000 (a large number) to allow convergence of the estimates.

clf_logistic2 = LogisticRegression(solver='sag', max_iter = 10000, penalty = 'l2').fit(X_train, np.ravel(y_train))

As in the above section we make predictions using the evaluation dataset.

preds = clf_logistic2.predict_proba(X_eval)

These predictions are stored in a dataframe instead of an array.

preds_df = pd.DataFrame(preds[:,1], columns = ['prob_accept_deposit'])
true_df = y_eval

Based on the same approach for selecting the best cut-off point, we implement the same algorithm to find the optimal probability cut-off point in order to balance our recall metric. Again we try to classify the probabilities using different cut-off points from 0 to 1 by increments of 0.001

numbers  = [float(x)/1000 for x in range(1000)]
for i in numbers:
    preds_df[i]= preds_df.prob_accept_deposit.map(lambda x: 1 if x > i else 0)
preds_df.head(5)

   prob_accept_deposit  0.0  0.001  0.002  ...  0.996  0.997  0.998  0.999
0             0.066219    1      1      1  ...      0      0      0      0
1             0.055719    1      1      1  ...      0      0      0      0
2             0.809286    1      1      1  ...      0      0      0      0
3             0.062395    1      1      1  ...      0      0      0      0
4             0.020000    1      1      1  ...      0      0      0      0

[5 rows x 1001 columns]

Then we calculate the metrics: accuracy sensitivity, and recalls for deposit and no deposit for various probability cutoffs.

cutoff_df = pd.DataFrame( columns = ['prob','accs','def_recalls','nondef_recalls'])
for i in numbers:
    cm1 = metrics.confusion_matrix(true_df, preds_df[i])
    total1=sum(sum(cm1))
    accs = (cm1[0][0]+cm1[1][1])/total1
    
    def_recalls = cm1[1][1]/(cm1[1][1]+cm1[1][0])
    nondef_recalls = cm1[0][0]/(cm1[0][0]+cm1[0][1])
    cutoff_df.loc[i] =[ i ,accs,def_recalls,nondef_recalls]
print(cutoff_df.head(5))

        prob      accs  def_recalls  nondef_recalls
0.000  0.000  0.110392          1.0        0.000000
0.001  0.001  0.110392          1.0        0.000000
0.002  0.002  0.110392          1.0        0.000000
0.003  0.003  0.110554          1.0        0.000182
0.004  0.004  0.110554          1.0        0.000182

Now we are able to choose the best cut-off based on the trade-off between deposit recall and no-deposit recall.

cutoff_df <- py$cutoff_df

names(cutoff_df)[names(cutoff_df) == "prob"] <- "Probability cut-off"
names(cutoff_df)[names(cutoff_df) == "accs"] <- "Accuracy"
names(cutoff_df)[names(cutoff_df) == "def_recalls"] <- "Deposit Recall"
names(cutoff_df)[names(cutoff_df) == "nondef_recalls"] <- "No-deposit Recall"

cutoff_df <- cutoff_df %>%
    gather(key = "metric", value = "value", -`Probability cut-off`)

ggplot(cutoff_df, aes(x = `Probability cut-off`, y = value, color = metric)) + geom_line(aes(linetype = metric)) + 
    ggtitle("Accuracy, Deposit Recall and No-deposit Recall") + scale_fill_discrete(name = "Metrics", 
    labels = c("Accuracy", "Deposit Recall", "No-deposit Recall"))

The optimal cut-off point is the following:

cutoff_df["diff"] = abs(cutoff_df.def_recalls - cutoff_df.nondef_recalls)
best_threshold = cutoff_df["prob"].loc[cutoff_df["diff"] == min(cutoff_df["diff"])]
best_threshold = best_threshold.iloc[0]
print(best_threshold)

0.525

Then we reassign the probability of accepting a deposit based on the optimal threshold.

preds_df['prob_accept_deposit'] = preds_df['prob_accept_deposit'].apply(lambda x: 1 if x > best_threshold else 0)

We can slightly compare how differ the estimations with the real values by the difference in deposits.

Count of estimated deposits by the logistic model:

print(preds_df['prob_accept_deposit'].value_counts())

0    4818
1    1360
Name: prob_accept_deposit, dtype: int64

Count of real deposits in our test set:

print(true_df['deposit'].value_counts())

0    5496
1     682
Name: deposit, dtype: int64

For further information it is necessary to check the classification report:

target_names = ['No-deposit', 'Deposit']
print(classification_report(y_eval, preds_df['prob_accept_deposit'], target_names=target_names))

              precision    recall  f1-score   support

  No-deposit       0.98      0.86      0.92      5496
     Deposit       0.43      0.86      0.57       682

    accuracy                           0.86      6178
   macro avg       0.71      0.86      0.74      6178
weighted avg       0.92      0.86      0.88      6178

We can see now a more balanced recall metric, however the values are lower than the previous one. We check the accuracy score the model as follows:

Finally, we check the confusion matrix

# Print the confusion matrix
matrix_3 = confusion_matrix(y_eval,preds_df['prob_accept_deposit'])
print(matrix_3)

[[4722  774]
 [  96  586]]

Now we check the accuracy of the model after assigning the optimal probability cut-off point:

accuracy_log_reg_2 = round((matrix_3[0][0]+matrix_3[1][1])/sum(sum(matrix_3)), 3)
print(accuracy_log_reg_2)

0.859

The accuracy of this model is lower than the previous one.

We are interested in evaluating our model based on the metric recall aka true positive rate for subscribing to a term deposits:

recall_deposit_log_reg_2 = round(matrix_3[1][1]/(matrix_3[1][1]+matrix_3[1][0]), 3)
print(recall_deposit_log_reg_2)

0.859

We can see a considerable improvement of 1% difference

#AUC
prob_deposit_log_reg_2 = preds[:, 1]
auc_log_reg_2 = round(roc_auc_score(y_eval, prob_deposit_log_reg_2), 3)
print(auc_log_reg_2)

0.927

So far our first model yields better returns in terms of accuracy, recall and AUC. This model however, since it is penalised, the estimates have been affected. Therefore we can assume that the first model has less risk of overfitting.

Reduced Logistic Regression Model

Our dataset has 63 independent variables, and many of these do not impact on the target variable. Many of this variables are called, noisy data. The occurrences of noisy data in data set can significantly impact prediction of any meaningful information. Many empirical studies have shown that noise in data set dramatically led to decreased classification accuracy and poor prediction results (Gupta and Gupta 2019).

In order to eliminate the noisy data in our training dataset, we will use the Recursive Feature Elimination (RFE) method which is a feature selection approach. It works by recursively removing attributes and building a model on those attributes that remain. It uses the model accuracy to identify which attributes (and combination of attributes) contribute the most to predicting the target attribute.

Our goal is not reduce our data to 1/3. Up to 20 variables.

logreg = LogisticRegression(max_iter = 10000)
rfe = RFE(logreg, 20)

rfe = rfe.fit(X_train, y_train)

print(list(zip(X_train.columns, rfe.support_, rfe.ranking_)))

[('age', False, 42), ('duration', False, 33), ('campaign', False, 21), ('pdays', False, 16), ('previous', True, 1), ('emp.var.rate', True, 1), ('cons.price.idx', False, 25), ('cons.conf.idx', False, 41), ('euribor3m', True, 1), ('nr.employed', False, 27), ('job_admin.', False, 24), ('job_blue-collar', False, 6), ('job_entrepreneur', False, 32), ('job_housemaid', False, 7), ('job_management', False, 5), ('job_retired', True, 1), ('job_self-employed', False, 3), ('job_services', False, 4), ('job_student', True, 1), ('job_technician', False, 30), ('job_unemployed', False, 17), ('job_unknown', True, 1), ('marital_divorced', False, 19), ('marital_married', False, 35), ('marital_single', False, 22), ('marital_unknown', False, 38), ('education_basic.4y', True, 1), ('education_basic.6y', False, 14), ('education_basic.9y', False, 20), ('education_high.school', False, 26), ('education_illiterate', True, 1), ('education_professional.course', False, 28), ('education_university.degree', False, 10), ('education_unknown', False, 12), ('default_no', True, 1), ('default_unknown', False, 9), ('default_yes', False, 44), ('housing_no', False, 18), ('housing_unknown', True, 1), ('housing_yes', False, 29), ('loan_no', False, 31), ('loan_unknown', False, 2), ('loan_yes', False, 39), ('contact_cellular', False, 13), ('contact_telephone', False, 23), ('month_apr', False, 43), ('month_aug', False, 15), ('month_dec', True, 1), ('month_jul', True, 1), ('month_jun', False, 40), ('month_mar', True, 1), ('month_may', True, 1), ('month_nov', True, 1), ('month_oct', True, 1), ('month_sep', True, 1), ('day_of_week_fri', False, 36), ('day_of_week_mon', True, 1), ('day_of_week_thu', False, 34), ('day_of_week_tue', False, 37), ('day_of_week_wed', False, 11), ('poutcome_failure', True, 1), ('poutcome_nonexistent', False, 8), ('poutcome_success', True, 1)]

The variables having showing True are the ones we are interested in. Now we select the variables we are interested in by running the following chunk of code:

col = X_train.columns[rfe.support_]
X_train_reduced = X_train[col]
X_eval_reduced = X_eval[col]

Now we can train our model with our training data with 19 variables.

clf_logistic3 = LogisticRegression(max_iter = 100000).fit(X_train_reduced, np.ravel(y_train))

As we the model 1, we will make predictions and look for the optimal cut-off point for classification.

preds = clf_logistic3.predict_proba(X_eval_reduced)
preds_df = pd.DataFrame(preds[:,1], columns = ['prob_accept_deposit'])
true_df = y_eval

We try to classify the probabilities using different cut-off points from 0 to 1 by increments of 0.001

numbers  = [float(x)/1000 for x in range(1000)]
for i in numbers:
    preds_df[i]= preds_df.prob_accept_deposit.map(lambda x: 1 if x > i else 0)
preds_df.head(5)

   prob_accept_deposit  0.0  0.001  0.002  ...  0.996  0.997  0.998  0.999
0             0.269626    1      1      1  ...      0      0      0      0
1             0.287275    1      1      1  ...      0      0      0      0
2             0.921524    1      1      1  ...      0      0      0      0
3             0.381851    1      1      1  ...      0      0      0      0
4             0.287227    1      1      1  ...      0      0      0      0

[5 rows x 1001 columns]

We create as many confusion matrices as cut-off and calculate the accuracy and recalls for each confusion matrix cut-off.

cutoff_df = pd.DataFrame( columns = ['prob','accs','def_recalls','nondef_recalls'])
for i in numbers:
    cm1 = metrics.confusion_matrix(true_df, preds_df[i])
    total1=sum(sum(cm1))
    accs = (cm1[0][0]+cm1[1][1])/total1
    
    def_recalls = cm1[1][1]/(cm1[1][1]+cm1[1][0])
    nondef_recalls = cm1[0][0]/(cm1[0][0]+cm1[0][1])
    cutoff_df.loc[i] =[ i ,accs,def_recalls,nondef_recalls]
print(cutoff_df.head(5))

        prob      accs  def_recalls  nondef_recalls
0.000  0.000  0.110392          1.0             0.0
0.001  0.001  0.110392          1.0             0.0
0.002  0.002  0.110392          1.0             0.0
0.003  0.003  0.110392          1.0             0.0
0.004  0.004  0.110392          1.0             0.0

cutoff_df <- py$cutoff_df

names(cutoff_df)[names(cutoff_df) == "prob"] <- "Probability cut-off"
names(cutoff_df)[names(cutoff_df) == "accs"] <- "Accuracy"
names(cutoff_df)[names(cutoff_df) == "def_recalls"] <- "Deposit Recall"
names(cutoff_df)[names(cutoff_df) == "nondef_recalls"] <- "No-deposit Recall"

cutoff_df <- cutoff_df %>%
    gather(key = "metric", value = "value", -`Probability cut-off`)

ggplot(cutoff_df, aes(x = `Probability cut-off`, y = value, color = metric)) + geom_line(aes(linetype = metric)) + 
    ggtitle("Accuracy, Deposit Recall and No-deposit Recall") + scale_fill_discrete(name = "Metrics", 
    labels = c("Accuracy", "Deposit Recall", "No-deposit Recall"))

With this information we are able to choose the best cut-off point:

cutoff_df["diff"] = abs(cutoff_df.def_recalls - cutoff_df.nondef_recalls)
best_threshold = cutoff_df["prob"].loc[cutoff_df["diff"] == min(cutoff_df["diff"])]
best_threshold = best_threshold.iloc[0]
print(best_threshold)

0.403

We can proceed to classify our model with the best cut-off point:

preds = clf_logistic3.predict_proba(X_eval_reduced)
preds_df = pd.DataFrame(preds[:,1], columns = ['prob_accept_deposit'])
true_df = y_eval

We implement the cut-off point in the discriminatory process

preds_df['prob_accept_deposit'] = preds_df['prob_accept_deposit'].apply(lambda x: 1 if x > best_threshold else 0)

After that, we can see the results from the classification report:

target_names = ['No-deposit', 'Deposit']
print(classification_report(y_eval, preds_df['prob_accept_deposit'], target_names=target_names))

              precision    recall  f1-score   support

  No-deposit       0.96      0.73      0.83      5496
     Deposit       0.26      0.74      0.38       682

    accuracy                           0.73      6178
   macro avg       0.61      0.74      0.61      6178
weighted avg       0.88      0.73      0.78      6178

This model is less robust, by decreasing the number of variables to the top 20 most important ones, the accuracy has dropped almost 10%. We can infer that this model is very sensitive to input changes, suggesting that it can be easily overfitted. This will be considered later on in further model implementations.

We analyse the confusion matrix of this model:

matrix_4 = confusion_matrix(y_eval,preds_df['prob_accept_deposit'])
print(matrix_4)

[[4032 1464]
 [ 179  503]]

Indeed there are more false negatives than in the previous models.

We calculate the accuracy of the model:

accuracy_log_reg_3 = round((matrix_4[0][0]+matrix_4[1][1])/sum(sum(matrix_4)), 3)
print(accuracy_log_reg_3)